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ABSTB ACT 

The first step in deyeloping a communication 
competency assessment instrument 4s determining whether there is 
conceptual ^support for such an instrument. Funding should also be 
determined, since, speaking skills must be assessed as a person 
speaks r a procedure that requires mere time and money tc complete 
than in the case of machine-scored tests\Next f specific skills must 
be identified.. This means a delineation of skills mere complete than 
the generic "speaking," "listening , " and "interpersonal" categories^ 
Once this is accompl ished, the search for instruments can begin. It 
must be kept in mind that the purpose of the instrument if it is to 
measure skill attainment, must be to elicit skills fcx subsequent 
measurement* Following pilot testing r the instrument _is ready for* 
refinement, which wi*th the evaluation process must continue until 
.some assurance is obtained that the instrument is valid and reliable. 
Follow-up procedures arg also* essential at this stage cf development* 
Test development can become a" futile endeavor if careful 
consideration is not given to all procedural elements thrcughcut the 
process. Accountability hinges on the aissumptioij, and ultimate proof r 
of -the consist^at nature* of purpose^ concept r and instrument- 
(HOD) 
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, ^EHE DEVELOPMENT AND REFINEMENT' OF 'A * ( 

, COMMUNICATION - COMPETENCY ASSESSMENT INSTRUMENT 

.'■ : . :'. \ ' ■ ! 

Competency-based education is receiving increased attention in bo£h 
higher* evocation and public 'school institutions (see Jaeger & Tittle, • 
1980). Accompanying this approach is the need to determine the con- 
stifuent elements- of competence and to develop' methods pf assessing these 
elements. Some models are available for adpption (see, for example, 
Bassett et al., 1978; Muchmore & Galvin, 1980). .Others (Backlund, et-al., 
1979) have developed instrument evaluation criteria. These' models and *$st 
development criteria are extremely helpful when one considers the 
foreign nature af test development to most speech communication faculty 
members. ' ' 

The purpose of this paper *is to examine* the process of developing 
a competence assessment instrument in the area of communication skills, 
to provide insight into issues facing a dest developer/ and to j^ovide 
examples' of how issues are dealt with and how test development progresses.. 
Figure 1 outlines the test development process. As indicated in the 

y * v 

figure, there are a number of considerations to be made before tests are 
adopted for use. Financial, conceptual,, and feasibility decisions 
must be made, titien tests ar>e examined or developed, and; finally, the 
instruments are refined. 

INITIAL CONSIDERATIONS 
The first issue faced* in the development process tends to be e 
conceptual in nature — Is there conceptual support ^or a competence 
assessment instrument? Do significant individuals believe in competence- 
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based education? And, if so, is the support official or tentative? 

The "wait- and see" attitude taken by many administrators can, in the long 

i 

run, become a major obstacle to implementation of any such project. 
♦ 

Official support, 'too, may be designed for the concept of competency- 
based education, but not for assessment procedures. In the area of r 
* • 

communication skills assessme^, individuals have cause for alarm. 
Opinions on the worth of a project are often based on the validity of the 
-instrument (not that validity isn f t .crucial) instead of on the coricept 
tttat persons should be able to show that their communication skills are 
sufficiently developed by a certain point in time to advance in their 
educational or occupational systems. Professionals, not in the communication 
field often feel that the development of communication skills is innate 
and thus they do not support the training ex^eriejiceft which must 
accompany* any competence-based program of assessment.. 

Funding is also essential. The promise o,f funding is not sufficient ' 
grounds for instrument development; tt*e process \is too long and the • * 

results are too tentative. In the case of communication skills .assessment, 

• j - . 

funding is a major issue since it is important that skills are measured, 
directly — i.e., that a speaking skill be assessed^as 'a person speaks. The 
ability to speak cannot validly be assessed through multiple-choice or 
'other pencil-^aper items. One is then assessing knowledge and not skill 
(behavior). Thus, assessment procedures naturally require more 
financial resources since the amount of time to complete the assessment 
is greater than in the case of machine-scored tests. The purpose of the 
assessment instrument, though, is an important determination in the 
funding process. 
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In the area of speech communication, instruments have a variety of 

purposes (see Larson et al.. , 1978'). Some tests are used as screens — to 

assess degree of reticence^ knowledge, skill-attainment — so that remedial 

programs can be ^established and students can achieve a theoretical level 

# i # # 

of competence. Others act as* barriers > to protfibit students from progres- 
sing., Such is the case with aptitude tests and exit exams. Those viewed 

as screening devices carry the additional burden of extensive support 

* • is 

program development — e.g., learning labs, remedial programs, etc. — specifically 
designed to increase student performance levels. The purpose of the 
example to be used throughout this paper — the development of a competence , 
instrument to assess demonstration of speaking and listening skills of 



college sophomores so that they may develop a level of competence to allow 



successful interaction in future coursework — was that o*f a screening detfi 
The main purpose of the competence program, though, is often mandated by 
the institution's attitudes and conceptual orientations. 

At this point in the test development process, the specific skills 
must be identified. This means a delineation of skills more complete than 
the generic, "speaking, 11 ''listening," and "interpersonal 11 categories often 
found in this field. For example, should* the student be able to express 
empathy," gesture at appropriate places while speaking, or listen for 
understanding? If so, will these need to be dissected further into 

; 

specific behavioral competencies? Or ^s a holistic approach* preferred? 
In the latter case^ increased emphasis must be placed ori the method of 

determining competence so that a reliable measure is utilized. 

i & , m 

Once the domain of specific competencies is identified, the searGh 

for instruments begins. Examination of the instruments must include 
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consideration of:. (1) the congruence of the theoretical approach to that 
developed by the institution (i.e., conceptual orientation); (2) the 1 * 
determination that the instrument is assessing the specific competence* 
<areas identified as in need of assessment (i.e., content validity); and ' 
(3) the conclusion that the testing method is consistent for the 
purposes and resources of the institution (i.e., practical i.tyO . , » 
The examination of these instruments should be an extensive process. 
Only through actually using the instrument can one determine the instru- 
ment's value. Concurrent validity checks cart also be employed. For 
example, an assessment instrument containing a variety of communication 
skills -^an be compared to holistic evaluations. conducted in ^classes^ or 
by other trained persons. Above all else, the instrument must.be feasible, 
reliable, a valid, and free from bias (Mead, 1980). If all predetermined 
needs are met, the instrument can>be adopted with the intent to gather 
normative data pn students to set specific competence levels and to monitor 
ttie instrument fc$r {/ossJLble refinement of assessment techniques to increase 
reliability^! ratings. If any of these needs — conceptual^ validity, 
practicality-- are not met, the instrument should not be adopted. .The 
institution should determine that the more worthwhile procedure would be 

£o develop the instrument to fulfill all existing conditions constituting 

> \ '' ' * , 

the original basis for assessing communication Competency (see Hambleton~& 

Eignor, 1980, for 'additional information on'test development). 

. j ^he remainder of this paper, then, will concentrate* on the process of 

developing and refining a specific assessment instrument — the Communication 

Competency Assessment Instrument (CC^l) . The development of this */ 

instrument has been detailed elsewhere (R.Rubin, in press; R.Rubin, 1981). 

* * 

What wijl be emphasized is the process of test development and decisions 



test developers must face along the way. 



3t^develop"ers must 
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TEST DEVELOPMENT ? 

, The development and refinement o€ a communication competence assess- 

ment instrument is a te'dious and sometimes frustrating^ process requiring 

•the ability to stand back from one's efforts and objectively evaluate 

the product^and the .courage to do so. Instrument refinement becomes the 

£ * 

,key to the creation of a valid instrument that can be used in a reliable 
manner (for additional information on reliabil^ty~and validity, see 
Standards , 1974). - 

The first, an<J most crucial area in test development is the identi- 
fication of the specific competericies to be assessed. .Without this stage, 

t the* instrument has no internal check procedure and* content validity^is 

* "* 
difficult to determine. In relation to the development of the CQAI, the 

competencies endorsed by the Speech Communication Association (Bassett, 

Whittington, & Staton-Spicer , 1978) were used as a framework;, these were 

extended to the specific context of concern (Educational) and new examples 

were created,' providing a sound basis for the - question development process 

Another available technique is the consensual agreement by a knowledgeable 

group of individuals as to the. specific skills one^whould possesS for the 

context of ooncern; this, by the way, occurred prior to the Speech 

I 

Communication Association's endorsement of the aforementioned competences 
and skills. THus , the broader picture of what con£titutes # competenc'e must 
underlie creation of skill areas to be tested. 

The method of assessment must then be geared to the skill* areas. 
That is, how best can one ass^^^a - particular- skill? The indirect 

. . . " V ■ • 

measurement of behaviors has been rightly criticized as an improper method 



in assessing sneaking and human relations skills (Stiggins, 1981). The 
writing and/or reading prc&ess should not, in any way, become a possible 
intervening variable. That is, one must be certain that when, for example, 
the abil-tfy to summarize messages % is assessed, it is only the students 1 oral 
summarizing abilities, and not writing abilities, that are of concern. 
Human relations skills can likewise be demonstrated. The assessment of 
listening skills is a difficult issue. Direct assessment of listening 
ability would, by definitidn, be assessed within the individual's 
receiving and decoding processes. This is somewhat impractical. Thus, 
an indirect method, through writing (or preferably, speaking) must be 
utilized. The -doubt, then, that a true assessment is occurring will 
always exist when understanding is filtered through the writing or 
speaking process. * 

j In addition, the assessment" method should assess only, that student's 
speaking and/or listening ability. Some instruments require one student 
to describe a geometric! figure while the other listens and draws what is 
"heard. 11 This interaction process confounds the test results. ""That is, 
the student^ who is listening may have a low skill level and cause the 
student who is describing to receive a lower score on descriptiQn ability 
than he/she would have received with a good-listener partner. 

In the direct assessment of speaking and listening skills, thfe 
use of recording materials is often an issue (A.Rubin, 1981). Does the 
videotape equipment add undue stress to the situation? How can we 
record for posterity all students 1 responses in the event thatT accounta-. 
bility becomes an issue? * Can inter-rater reliability counter the excessive 
costs ofl^capturing students' responses on tape? . , ' 
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As implied previously, the test mofte can viably be mandated by the 
inavailability of financial resources. This being the case, the test 
results of indirect measurement instruments must be carefully examined 
for possible invalidity. A more preferable route would be to obtain 
funds to sponsor the most* viable, and valid instrument possible. 

The assessment materials in a coinmunj.cation competency assessment 
instrument, be they questions or statements to which students respond, 
are then, created^, Two important points must be kept in mind,, in this 
regard. First, the purpose of the instrument, if it is to measure skill 
attainment (and not knowledge of coilcepts or material), must be to elicit 
skills for subsequent measurement. Second, the context in which the 
materials are based must be kept, relevant to students—i.e . , they must 
have the experience necessary to show the skill. 

An example from the CCAI demonstrates both of these principles. 
The competence area dealt with .oralv message evaluation. Specifically, • 
the skill was^the ability to describe another'^ point of view when Ithe 
other disagreed on an issue. The context 'was identified as a course which 
the student either liked or disliked ^ery much. StudetLts had no difficulty 
identifying for themselves such a course. Ttiey were then a^ked to 
describe the viewpoint of one of their fellow classmates who had the opposi 
viewpoint. This did not require that they would have necessarily heard 
others make such descriptions (although that was possible), but merely 
to describe what an opposing viewpoint would 'be without contaminating the 
description with one's owtP view about *the clas'si The rating, of responses 
(on a 5 point scale) ranged from 'complete, objective description of the 
other's viewpoint to complete^ subjective description of the student's . 
own view^botft the class. The level of 'acceptable "competence" at the 



midpoint was marked by .a semi-objective description of the other's 
view with possible inclusion of a brief statement about the -areas of 
contrast; the opposing viewpoint needed to be well-representted in the 

V J 

description. All in all, the item Was based on the students 1 basic 
skills of describing their experiences. , 

The creation of- the skill levels and rating instrument follows an 
examination of the assessment materials and their validity. Are the 
materials, in actuality, assessing what t'hey are supposed to? Consul- 
tation with professionals .in the area can help determine face validity. 
A better method might be blind review of materials by these consultants, 
asking each to either .identify or match the materials to the original 
list of competencies. This provides some assurance that the development \ 
of the instrument is progressing in a 5 positive^directi^n. * 

Methods of assessing competent possession of these skills must then 

be formulated. It is generally believed that a "competent—not competent" 

/ 

judgment is too restrictive for most skill areas. A wider range taf I 
responses is a preferable format (as in the example above). At times this 
may prove problematic. This *was the case, for example, in the CCAI 
competence of "distinguishing facts from opinions/ 1 Students were to 
indicate if a previously-heard statement was based on fact§- or on opinions . 
It would have been much simpler to score the* students 1 responses as either ^ 
"right 11 or "wrong" than to create a % 5-point range of responses, as was done. 
The range, though, disprove an adequate method of assessment and inclusion 
of the/ range added to* the consistence of rating format, 'in other words; 
if nothing is lost b^Soaintaining consistency, tfte range of responses is * 
preferable to a Mf competent^Wiot competent" judgment. 
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The rating instrument, then, must be clear enough that it can be 

used in a unifort^ manner by all raters. A team of persons similar in 

composition to those who will use the instrument (e.g., faculty) can be 

helpful ^Ln refining the rating instrument. Inter-rater reliability tests 

* ♦ 

help to determine the sufficiency and depth of description for the skill"' 

* levels. Refinement of assessment materials and level descriptions can, 

in actuality^ continue for a long period of time, depending on the 
f 

level of description foun3 in the rating book. 

The instrument is then ready for pilot testing. Students may be 

chosen to participate on the basis of pre-determined skill levels; in thi^s 

case, raters should be unaware. of the Students', reputations.- Students 
» * * # 

• ■ \ 

could also be chosen at random to assure a representative sample of 
possible responses. ^This is, perhaps", preferable, especially when assessing 
the reliability of. the instrument. 7 The validity of the test can also be 
determined T?y use of concurrent assessments of skills. 

' * REFINEMENT OF THE INSTRUMENT - . 

Instrument refinement^ then, takes the 'form of examining students' 

( 

responses to the questions as they are used (i.e., do Jthe students 
understand what is being asked of them?) and the distribution of responses 
on the rating levels. In the former, raters often indicate if the 
instructions are clear and ,if students understand what they are asked to 
» t do. * In the latter instance, pne may find that all students receive the 

same score on an item. When the CCAI was being developed, one question- ^ V 
> r / asked students to use a chronologiqal order to explain all of their 

activities during the day. . Almost al^studeiits correctly used this order % 

\*y , and the coefficient alpha analysis rejected the item on the* grounds that . 

: 4' ' * . 

eric ' ' - 11 - 
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it was not discriminating a^ng skill levels as well as o^her items, 
Coefficient alpha is a measure of internal consistency of A test; th^t is, 

0 \ 

a student who scores high on identifying a main point in a lecture should 
also score high on understanding* Both of these are listfeninV competencies. 
This measure of internal consistency has purported importance, in test 

ne when 



development lieterature (Cronbach, 1951), but is somewhat troublesome 



one does not view the ability to* communicate as a hqjlistic concept. 

If one assumes,* for example, that human relations skills are not ^ 

necessarily connected to listening skills which are not necessarily 

connected to nonverbal skills ,__£hen the coefficient alpha indication would 

be meaningless. Thus, a perfect intercorrelation of responses is not 

ideal^. An alpha of .80 is respectable, especially for instruments with 

.les^than 30 items (Nunnally, 1978). m ' % 4 

After a period of refinement and elimination of inconsistent items, 
t 

^fhe instrument sHould again be piloted" on a randomly chosen group of * 
students/ prototypes of those who will be assessed in the future. This 
refinement and £valyation process must continue until some assurance that 

■ the instrument is valid and reliable is obtained. This also helps in the 
establishment of norms and determination 6 f* competence levels used to 

r identify students in need of remedial instruction. As mentioned 

previously, either self-paced or classroom instruction tnust be in place 

/ 

once the * instrument is adopted or the purpose of the instrument reverts 
to that of a barrier to" a student ^s progress. Piloting- the instrument 
on students at a number of institutions increases assurance that one s 
own population is not distinctly different from the entire population for 
which die instrument is intended/* The bias-free quality of the instrument * 

"12 
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is of utmost importance and can be assessed by wide-spread use throughout 
a large population. 

Follow-up procedures are also essential at this stage of development'. 

^If an instrument is created; for instance, tio assess students' communication 

skjill attainment at the sophomore level to determine the -need for instruction 

before upper-level courses are attempted, both implications must be 

examined. First, do students taking upper-level courses' actually need a ' • 

higher level of skill achievement? And second, do studentsVwho do well 

in those courses have superior communication skills? What this analysis 
« 

demonstrates is the interactive nature of concept/ purpose, and instrument. 

Any inconsistency among the three can be an indication of an invalid . 
instrument .or testing procedure. Accountability hinges on the" assumption, 
..and ultimate proof, of the consistent nature of purpose, concept, and 

instrument.-;^' ^ * * * 9 

The implication throughout^this paper tfeat this process of developing 
and refining a communication competence assessment instrument is open to 
consternation at multiply points is, indeed, accurate. Test development 
can become a futile endeavor if careful consideration of all procedural 
elements is not x accomplished throughout the process. , And even then, * 
assurance is not guaranteed. State legislatures, governing boards, and 
even one's own colleagues can create stumbling blocks along the way. 
Initial attitudes of support can quickly change into that which is so 
often heard in this profession—that .communication skills are innate, they 
need not be taught. Or perhaps what will emerge is the, attitude that 
students only need listening skills in classrooms; professors speak and 
students listen. This bias is among us and must be considered above &11 else. 
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The future of communication competence assessment, though, is 

promising. A number of persons at local, state, and national levels are 

* 

being heard, listened to, and are even persuasive about remedying this 
past oversight in basic skills education. Instruments being developed 
must be the best that they can be to increase the credibility and strength 
of the .competency-based' approach and the profession at large. 
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FIGURE 1 

Flow Chart of Test Development Process 
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